Evaluation of Japanese-Chinese MT System using AAMT’s Test-Set

The methods for evaluating the quality of a machine translation (MT) system have two major problems. One is the establishment of criteria which are independent of evaluator objectivity and the other is reducing man-hour costs needed to evaluate huge sets of translation sentences. With these aims, JEIDA (Japan Electronic Industry Development Association) had developed JEIDA’s Test-Sets [1] which are applied to English-Japanese and Japanese-English translation evaluation. One of the features of their Test-Sets is translation examples that include grammatical checkpoints, which are answered “Yes” or “No” by evaluators. Compared with conventional evaluating methods, such as grading translation on its quality into 3 or 5 levels, the method based on answering yes/no questions is more objective. Since 2007, we have been developing the new Test-Set for evaluating quality of Japanese-Chinese MT systems, expanding JEIDA’s Test-Sets [2]. In June 2009, our new Test-Set was completed and made publicly available [3]. The work described in this paper has been developed by the Special Working Group of AAMT (Asia-Pacific Association for Machine Translation) (we call our Test-Set “AAMT’s Test-Set”). In 2008, we evaluated the quality of several actual Japanese-Chinese MT systems, applying AAMT’s Test-Set. In order to compare with conventional methods, we also carried out subjective evaluation focused on fluency and adequacy. In this paper we would like to describe the merits of the evaluation method using AAMT’s Test-Set comparing with conventional methods. In Section 2, we will overview AAMT’s Test-Set showing examples of yes/no questions. Section 3 describes the details of evaluation experiments we carried out. Chapter 4 describes variability of evaluation results depending on evaluators, and analyzes correlation among evaluating methods such as fluency evaluation, adequacy evaluation, yes/no evaluation and statistical evaluation (BLEU)[4]. Moreover, we will briefly discuss strong and weak points of MT systems, comparing Japanese-Chinese and Japanese-English MT systems.